Not-So-Great Expectations

In my teaching and consulting work, I often ask people how they recognize a problem, and they often offer “inconsistency with expectations” as one way to do it. I agree, but I also think we should be careful to think things through. A product that fulfills our expectations may not be satisfying, and a product that violates our expectations may be terrific.

Several years ago, I bought a new computer for the kids. The discount retailer offered only Windows Vista as the operating system. I had already heard plenty about Vista, and I considered going elsewhere. However, I figured that it might be a good thing to have one computer with Vista in the house for testing purposes—and for the kids, it would Build Character.

In this house, “Daddy” also means “technical support”, so I frequently found myself troubleshooting and configuring the kids’ system. I was a Windows XP user at the time. As usual, Microsoft had run its User Interface Scrambling and Feature Hiding tool just before Vista’s general release, so I couldn’t find the controls and settings I wanted to find. Moreover, every time I tried to something, Vista would interrupt me, pop up a dialog, and ask if I had really asked to do that something. If Vista had been an employee, I would have fired it on the first day. “Yes, I did want to do that. Why do you keep asking me? Yes, I am who I say I am. Why do you keep asking me that? Don’t you think if I weren’t who I said I was, I would simply lie to you and answer Yes?” I expected Vista to suck. And it sucked in all the ways I had expected, and more. But it was consistent with those expectations.

More recently, I purchased a Nest thermostat. My expectations weren’t met; they were exceeded. I was expecting to have to fetch a screwdriver to install it; nope, the unit comes with a screwdriver (both slot and Philips head). I was expecting to have to do the fiddly wrap-wires-around-a-terminal thing. When I opened the package, the thermostat didn’t have screw terminals, as I had expected; it had clearly labeled, spring-loaded, easy-to-use wire clips. I was expecting that I’d have to patch and paint where the big old thermostat had been; the Nest is small and round. No problem; the Nest comes with a backing plate that’s slightly larger than my old unit’s backing plate. I was expecting to have to connect to my computer or do some other elaborate setup stuff. Nope; the unit automatically found our wireless router, asked me for the access key, and connected itself to the Web. So the product wasn’t what I expected, but that wasn’t a problem.

Expectations typically refer to the future, but many things that we might call “expectations” in casual speech are not clear in advance. They’re subconscious, tacit, or constructed on the fly. The set of expectations that we could have about something is intractably large, and it would be cognitively difficult and practically expensive to make them explicit and to note them—and for many problems, it wouldn’t be necessary to do that, either. Here’s an example. Look closely, and ask “what’s wrong with this picture?”

In my version of Microsoft Outlook 2010, I use the search function to find an email message that has a particular string in it. The search returns no results for the current folder, so Outlook offers to search all mail items. In the search result, the fields for the sender, the subject of the message, and the folder all have the last characters truncated.

Neat bug, huh? Now, I could say that I recognized a problem because the behavior was inconsistent with my expectations, and you might say the same. Yet I doubt that we would ever have bothered to express this expectation: “I expect all data fields to display all characters, even the last one”. It might be more accurate to say that, when I spotted the bug, I constructed the expectation in the moment, in response to an observation of a violation of some pattern or consistency heuristic.

Our expectations, desires and requirements for a product aren’t static. We develop them as we develop the product, as we become familiar with it, and as we develop our relationship with it. When people say that something is a bug because it’s inconsistent with their expectations, I think they might be confusing expectations with desires. That is, when people say “that’s not what I expected,” they really mean “that’s not what I want“. Inconsistency with expectations isn’t a big deal at all when people are surprised or delighted by the product. In some cases, inconsistency with expectations might cause temporary discomfort or disorientation, but after a bit of experience and familiarity, what we have might turn out to be better than what we expected. Moreover, someone might have a set of expectations fulfilled, and upon observation and reflection they might realize that what they expected was not what they wanted after all. As oracles—means by which we identify a problem during testing—an inconsistency with history or expectations might be superseded by a consistency with purpose.

Every moment that we’re testing, we have the capacity to recognize a new risk, a new problem, or a buried assumption, even without an explicit expectation. Humans can do this at the drop of a hat, but machines cannot. This is why it’s so important to have humans testing, interacting with the product, observing and experimenting at all levels and with all of the product’s interfaces. By all means use tools to help you, but beware The Narcotic Comfort of the Green Bar! Operate and observe the product directly, and re-evaluate your checks and their outputs from time to time. Don’t just ask, “Is this what we expect?” Ask also, “Is this (still) what we want?”